Learning Boolean Functions via the Fourier Transform

We survey learning algorithms that are based on the Fourier Transform representa tion In many cases we simplify the original proofs and integrate the proofs of related results We hope that this would give the reader a complete and comprehensive un derstanding of both the results and the techniques Introduction The importance of using the right representation of a function in order to approximate it has been widely recognized The Fourier Transform representation of a function is a classic representation which is widely used to approximate real functions i e functions whose inputs are real numbers However the Fourier Transform representation for functions whose inputs are boolean has been far less studied On the other hand it seems that the Fourier Transform representation can be used to learn many classes of boolean functions At this point it would be worthwhile to say a few words about the Fourier Transform of functions whose inputs are boolean The basis functions are based on the parity of subsets of the input variables Every function whose inputs are boolean can be written as a linear combination of this basis and coe cients represent the correlation between the function and the basis function The work of LMN was the rst to point of the connection between the Fourier spec trum and learnability They presented a quasi polynomial time i e O n log n algo rithm for learning the class AC polynomial size constant depth circuits the approximation is with respect to the uniform distribution Their main result is an interesting property of Computer Science Dept Tel Aviv University This research was supported by THE ISRAEL SCIENCE FOUNDATION administered by THE ISRAEL ACADEMY OF SCIENCE AND HUMANITIES the representation of the Fourier Transform of AC circuits based on it they derived a learn ing algorithm for AC For the speci c case of DNF the result was improved in Man In Kha it is shown based on a cryptographic assumption that the running time of O n log n for AC circuits is the best possible In AM polynomial time algorithms are given for learning both probabilistic decision lists and probabilistic read once decision trees with respect to the uniform distribution In this paper we concentrate on deterministic functions namely deterministic decision lists hence some of the techniques and the results of AM do not appear here The work of KM uses the Fourier representation to derive a polynomial time learning algorithm for decision trees with respect to the uniform distribution The algorithm is based on a procedure that nds the signi cant Fourier coe cients Most of the work on learning using the Fourier Transform assumes that the underline distribution is uniform There has been a few successful attempts to extend some of the results to product distributions In FJS it is shown how to learn AC circuit with respect to a product distribution In Bel the algorithm that searches for the signi cant coe cients is extended to work for product distributions However in this paper we concentrate on the uniform distribution There are additional works about the Fourier Transform representation of boolean func tion The rst work used Fourier Transform to the show results in theoretical computer science was the work of KKL that proves properties about the sensitivity of boolean functions The relation between DNFs and their Fourier Transform representation is also studied in BHO Other works that are investigating the Fourier Transform of Boolean functions are Bru BS SB In this work we focus on the main results about the connection between Fourier Transform and learnability The survey is mainly based on the works that appeared in LMN AM KM Man Some of the proofs are a simpli cation of the original proofs and in many cases we try to give a common structure to di erent results Some of the learning results shown here are based on the lower bound techniques that were developed for proving lower bound for polynomial size constant depth circuit Ajt FSS Yao Has When we need to apply those results we only state the results that we use but do not prove them The paper is organized as following Section gives the de nition of the learning model and some basic results that are used throughout the paper Section introduces the Fourier Transform and some of its properties Section establishes the connection between the Fourier Transform and learning This section includes two important algorithms The Low Degree algorithm that approximates functions by considering their Fourier coe cients on small sets and the Sparse algorithm that is based on the work of GL KM which learns a function by approximating its signi cant coe cients In Section we show various classes of functions that can be learned using the above algorithm We start with the simple class of decision lists from AM We continue with properties of decision trees from KM The last class is boolean circuits there we show properties for both DNF and AC circuits from LMN Man Preliminaries Learning Model The learning model has a class of functions F which we wish to learn Out of this class there is a speci c function f F which is chosen as a target function A learning algorithm has access to examples An example is a pair x f x where x is an input and f x is the value of the target function on the input x After requesting a nite number of examples the learning algorithm outputs a hypothesis h The error of a hypothesis h with respect to the function f is de ned to be error f h Pr f x h x where x is distributed uniformly over f gn We discuss two models for accessing the examples In the uniform distribution model the algorithm has access to a random source of examples Each time the algorithm requests an example a random input x f gn is chosen uniformly and the example x f x is returned to the algorithm In the membership queries model the algorithm can query the unknown function f on any input x f gn and receive the example x f x A randomized algorithm A learns a class of functions F if for every f F and the algorithm outputs an hypothesis h such that with probability at least